Skip to content

Deterministic self-healing drift: remove the LLM from remediation#335

Merged
jpr5 merged 7 commits into
mainfrom
drift-rearch-clean
Jul 23, 2026
Merged

Deterministic self-healing drift: remove the LLM from remediation#335
jpr5 merged 7 commits into
mainfrom
drift-rearch-clean

Conversation

@jpr5

@jpr5 jpr5 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Deterministic self-healing drift — remove the LLM from remediation

The drift auto-remediation subsystem broke ~8 times because it let an LLM freewrite edits to the detector's own fixtures and then tried to police that with a 916-line anti-cheat predicate — an unwinnable loop. This replaces that whole loop with robustness-by-construction: most drift now self-heals with no fix at all, mechanical model-family churn is synced deterministically, and the human surface shrinks to exactly two real decisions (mock a brand-new family or not; follow a genuine protocol redesign or not).

What changed

  • The LLM is gone. scripts/fix-drift.ts (the freewriter) and scripts/drift-success-predicate.ts (the 916-line predicate) are deleted, along with their now-orphaned tests.
  • Deterministic remediation. A daily cron fetches live /models and either:
    • ok-applied — mechanically edits model-registry.ts (via the TypeScript compiler API, not text munging) to drop a deprecated family → opens a PR.
    • needs-human — a brand-new/unclassified family, a still-referenced deprecation, or a registry structural mismatch → writes a drift-proposals/ note and opens a PR where a human sets Decision: include; the next run applies it.
    • Both PR paths are gated by scripts/drift-sync-check.ts (changed-file allowlist + classification checksum-pin re-assert + clean re-collect, all fail-closed) and are idempotent across daily re-fires (deduped on a stable content-hash changeset key). No auto-merge — humans merge.
  • Fixes the recurring failure at the root. The collector now isolates per-leg failures, so one bad live leg can no longer quarantine the batch and poison every PR's base drift report.
  • Self-healing live legs. 6 legs (anthropic, openai-chat, openai-responses, bedrock, ws-responses, ws-gemini-live) discover a live model and honest-skip on infra/deprecation instead of hard-failing, via shared resolveLiveModel/isInfraSkip helpers.
  • Frozen classification rules are checksum-pinned so a silencing edit trips a loud test failure.

Review

Built as 14 parallel slots, then a Tier-3 review that found and fixed 8 mandatory defects — each with a local red-green proof, each re-reviewed to zero on the fixed state. Notably: the sync core's recollect gate was silently breaking the new-family human-routing path (and a fake test was hiding it), and the workflow didn't persist the needs-human note (so the human-decision path was unreachable). Both would have shipped green otherwise.

Test evidence

tsc --noEmit 0 errors · full suite 4766 passed / 0 failed · build ok · classification pin 14/14 · workflow tests 37/37.

Fast-follow (non-blocking, not in this PR)

3 now-inert helper functions to delete, a couple of dead workflow permissions, excludeFamilies data not yet pinned, the byte-mirror not equivalence-tested, claude-fable-5 re-proposed each run, and a pre-existing Bedrock $metadata shape gap (only surfaces under real AWS creds).

🤖 Generated with Claude Code

https://claude.ai/code/session_01KamK73Wu5heLxJEogM6hHC

jpr5 added 6 commits July 22, 2026 23:34
…tion to deterministic sync

The fix-drift LLM-in-the-loop remediation path (scripts/fix-drift.ts) and its
anti-cheat success predicate are removed. Remediation for drift now runs
through the deterministic model-family sync (drift-sync.ts /
drift-sync-check.ts introduced in the prior commit): known model-family
changes are applied programmatically, anything outside that scope is flagged
with a needs-human note, and a human still reviews and merges. The
fix-drift workflow, Slack summary, and shared drift-types are updated to
call the new sync path instead of invoking an LLM to freewrite a fix.
@pkg-pr-new

pkg-pr-new Bot commented Jul 23, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@335

commit: 07b1035

…re cases

The `drift-live-pr` CI check quarantined the whole batch as an unparseable
exit-5 failure:

    expected 'claude-haiku-4-5-20251001' to be 'claude-haiku-5-1-20260201'

Root cause is cross-block pollution of `resolveLiveModel`'s per-key memo, not
a model-pin mismatch. Both sides of the live-leg 3-way comparison already use
the same dynamically-resolved id. But when ANTHROPIC_API_KEY is armed (CI),
the live `describe.skipIf` block runs FIRST, resolves the real still-live id,
and memoizes it under the shared "anthropic" key — with no afterEach reset,
exactly like the real live block. The fixture self-healing block reset the
memo only in afterEach, so its FIRST case inherited the leaked live id instead
of resolving against its own stub, then asserted a raw `.toBe()` on model ids.
That AssertionError is not a structured drift report, so the collector could
not parse it (exit 5, fatal quarantine) rather than a non-fatal exit-2 report.

Fix: reset the memo in `beforeEach` of the fixture block so each case resolves
against its own stub regardless of what the live block left cached. Add a
seeding regression guard that simulates the armed-key live block populating the
memo first (no credentials needed), reproducing the CI ordering deterministically.

Siblings audited: ws-gemini-live isolates via a distinct cache key
(gemini-live-text-unittest); openai-responses/ws-responses have no fixture
block reading the memo after their live block; openai-chat has no such
assertion. anthropic was the only leg sharing one memo key across both blocks.

Local red-green (fixture-driven; live keys unavailable locally, so the true
confirmation is the re-run drift-live-pr CI):
  RED  (before fix): REGRESSION case fails with the verbatim CI assertion.
  GREEN (after fix): all 11 runnable anthropic cases pass (6 live-key skipped).
Full gate: tsc 0, eslint/prettier clean, unit 4722 passed (logic-pin 14,
fix-drift-workflow 37), drift 118 passed, tsdown build OK.
@jpr5
jpr5 merged commit 2203982 into main Jul 23, 2026
29 checks passed
@jpr5
jpr5 deleted the drift-rearch-clean branch July 23, 2026 06:55
jpr5 added a commit that referenced this pull request Jul 23, 2026
…r-equivalence guard, trim workflow perms (#336)

## Drift subsystem cleanup

Fast-follow to the deterministic self-healing drift re-architecture
(#335). Four low-risk cleanups the CR flagged as follow-ups; no behavior
change to the shipped remediation flow.

### What changed
- **Prune inert code** — the workflow now builds its PR body inline and
commits via `commitSyncChanges`, which left `buildPrBody`,
`gatedCommitFiles`, `isProductionFile` (and their orphaned helpers
`truncateBody`/`GH_BODY_MAX`/`GH_BODY_SAFE_MAX`/`affectedSkillSections`/`BUILDER_TO_SKILL_SECTION`)
with no production caller. Removed them and their tests.
- **Stop re-proposing `claude-fable-5`** — the registry intentionally
carries a forward-looking family that isn't live yet, so the sync
proposed removing it every run. Added a small `isForwardLookingFamily`
allowlist applied at the sync-policy layer
(`detectDeprecatedFamiliesForSync`); the canonical detector is
unchanged, so genuine retirements are still detected.
- **Pin `excludeFamilies`/`includeFamilies` data** — `logic-pin.test.ts`
previously froze only the classification *rules*, not the membership
data; a family could have been silenced by adding it straight to
`excludeFamilies`. Added a `DATA_FROZEN` checksum block (hashes the live
sets) that fails loudly on any membership change.
- **Sync-mirror equivalence guard** — added a test that verifies the
sync core's mirrored predicates stay faithful to the canonical
`detectDeprecatedFamilies` for normal families, and explicitly asserts
the one *intended* forward-looking divergence (so removing the filter
above fails the test).
- **Trim dead workflow permissions** — removed `checks: read` /
`statuses: read` (and their app-token counterparts + a stale comment)
from `fix-drift.yml`; nothing in the workflow consumes check-runs or
statuses, and it does no auto-merge.

### Review
Tier-1 CR (correctness + test-guard quality), 2 reviewers, **0 mandatory
findings** — both empirically verified the new pins/guards are
non-vacuous (recomputed all 6 data-pin hashes; confirmed the
mirror-divergence test fails if the forward-looking filter is removed).

### Test evidence
`tsc --noEmit` 0 errors · full suite green · `logic-pin` 20 ·
`drift-sync-mirror-equivalence` 28.

### Deferred follow-ups (not in this PR)
- **Bedrock `$metadata`**: the 2 forced-credential failures in the
bedrock drift leg are a *category error in the drift test's own
SDK-shape stubs* — they compare the SDK client-output shape (which
includes `$metadata`) against the raw wire body. `$metadata` is
synthesized by the AWS SDK from HTTP status/headers after
deserialization; it is **not** part of the Bedrock wire response. The
correct fix is to the test stubs (dormant — needs a live Bedrock
recording), not the mock.
- The mirror-equivalence test imports `models.drift.ts`, so a local
`pnpm test` run *with* provider API keys exported would also execute the
live-availability suites (harmless in CI, which sets no keys).
De-coupling would require extracting the canonical predicates into a
non-`.drift.ts` module — deferred.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KamK73Wu5heLxJEogM6hHC
jpr5 added a commit that referenced this pull request Jul 24, 2026
## Needs a human decision (drift-sync)

The deterministic, zero-LLM drift-sync found a model-family change it
must
NOT auto-apply (a genuinely new/unclassified family, a still-referenced
deprecation, or a registry structural mismatch). It wrote the note
file(s)
below and opened this PR so the decision is REACHABLE in the repo.

> **Not auto-merged — a human decides.**
>
> ⚠️ **The `Decision: include` instructions in this template do NOT
apply to this
> PR.** That protocol is scoped to a *new-family* note only:
`renderProposalNote`
> emits the `## Decision` block solely under `kind === "new-family"`,
and
> `parseProposalDecision` has exactly one call site — the addition half
of
> `runDriftSyncCore`. The deprecation half never reads a decision at
all.
>
> Both notes in this PR are **still-referenced deprecations**, and they
contain
> **no `Decision:` line whatsoever**. Writing one would be a no-op.
There is no
> marker, of any value, that makes a later run mechanically resolve this
note
> class — only a human registry edit can. **That edit is now part of
this PR**
> (see *Human decision* at the bottom), so there is no two-run hand-off
here:
> merging this PR both records the decision and resolves the drift.
>
> (For reference, the original template text: for a *new-family* note
you would
> set `Decision: include` to classify it, or delete the note / close the
PR to
> reject, then merge — and the next run would apply the registry edit
itself.)

### Note file(s) in this PR

<!-- drift-changeset: e08d8fcc953752d5 -->
- `drift-proposals/gemini-gemini-1.5-flash-deprecated-referenced.md`
<!-- drift-proposal-note:
drift-proposals/gemini-gemini-1.5-flash-deprecated-referenced.md -->
- `drift-proposals/gemini-gemini-1.5-pro-deprecated-referenced.md`
<!-- drift-proposal-note:
drift-proposals/gemini-gemini-1.5-pro-deprecated-referenced.md -->

### drift-sync outcome

```
npm warn Unknown project config "minimum-release-age". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options.
npm warn Unknown project config "block-exotic-subdeps". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options.
needs-human note(s) written this run — routed to human without a live re-collect (no registry edit to re-verify)
  [needs-human-still-referenced] gemini/gemini-1.5-flash: "gemini-1.5-flash" is deprecated but still referenced in source — routed to human (drift-proposals/gemini-gemini-1.5-flash-deprecated-referenced.md)
  [needs-human-still-referenced] gemini/gemini-1.5-pro: "gemini-1.5-pro" is deprecated but still referenced in source — routed to human (drift-proposals/gemini-gemini-1.5-pro-deprecated-referenced.md)
  [skipped] anthropic: live /models listing too short to trust for anthropic (10 raw id(s), need >= 19 — the number of families aimock mocks for this provider) — never mass-removing off a truncated or empty listing
npm warn Unknown env config "block-exotic-subdeps". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options.
npm warn Unknown env config "minimum-release-age". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options.
npm warn Unknown project config "minimum-release-age". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options.
npm warn Unknown project config "block-exotic-subdeps". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options.
^[[33m[STARTED]^[[39m Backing up original state...
^[[32m[COMPLETED]^[[39m Backed up original state in git stash (c62c488)
^[[33m[STARTED]^[[39m Running tasks for staged files...
^[[33m[STARTED]^[[39m package.json^[[0;90m — 2 files^[[0m
^[[33m[STARTED]^[[39m *.{ts,mts,js,mjs,cjs,json,html,css,md}^[[0;90m — 2 files^[[0m
^[[33m[STARTED]^[[39m *.{ts,mts,js,mjs}^[[0;90m — 0 files^[[0m
^[[33m[SKIPPED]^[[39m *.{ts,mts,js,mjs}^[[0;90m — no files^[[0m
^[[33m[STARTED]^[[39m prettier --write
^[[32m[COMPLETED]^[[39m prettier --write
^[[32m[COMPLETED]^[[39m *.{ts,mts,js,mjs,cjs,json,html,css,md}^[[0;90m — 2 files^[[0m
^[[32m[COMPLETED]^[[39m package.json^[[0;90m — 2 files^[[0m
^[[32m[COMPLETED]^[[39m Running tasks for staged files...
^[[33m[STARTED]^[[39m Applying modifications from tasks...
^[[32m[COMPLETED]^[[39m Applying modifications from tasks...
^[[33m[STARTED]^[[39m Cleaning up temporary files...
^[[32m[COMPLETED]^[[39m Cleaning up temporary files...
[fix/drift-2026-07-24-30074183288 85fc683] fix(drift-sync): mechanical model-family sync (needs-human note file(s))
 2 files changed, 14 insertions(+)
 create mode 100644 drift-proposals/gemini-gemini-1.5-flash-deprecated-referenced.md
 create mode 100644 drift-proposals/gemini-gemini-1.5-pro-deprecated-referenced.md
reason=needs-human
changeset-key=e08d8fcc953752d5
```

---

## Human decision: keep mocking these families

**Decision: KEEP.** `gemini-1.5-pro` and `gemini-1.5-flash` are gone
from Gemini's
live `/models` listing, but aimock still mocks them on purpose — clients
pinned to
older SDKs and older CopilotKit configs still send those ids, and
`isReasoningModel()` must keep answering `false` for them.

So nothing is being removed. Instead, commit `a3dc250` records the
decision in the
registry by moving both families from `includeFamilies.gemini` into
`excludeFamilies.gemini`'s existing **"Retired / legacy specialty"**
bucket, and
re-pins the two affected `DATA_FROZEN` membership hashes.

`excludeFamilies` means *"not counted as text-generation **drift**"*,
not
*"not mocked"* — `gemini-pro`, `gpt-3.5` and `gemini-2.0-flash-exp`
already live
there while being mocked and heavily referenced in `src/`. Prior art for
exactly
this move: #315 (`fix(drift): classify bare chat-latest alias in openai
excludeFamilies`).

Because `detectDeprecatedFamiliesForSync` draws its candidate set from
`includeFamilies[provider]`, the two families can no longer become
deprecation
candidates — drift-sync stops re-flagging them needs-human, which ends
the daily
red job and the daily Slack page. The two note files stay in this PR as
the audit
trail for the `excludeFamilies` entries; they become inert on merge and
will never
be re-read or re-written.

**Nothing that references `gemini-1.5-*` was touched.**
`src/model-utils.ts:54`
(`NONREASONING_FAMILIES`), the three gemini reasoning-capability suites,
the
`gemini.test.ts` / `vertex-ai.test.ts` conformance assertions and the
recorded
`sdk-shapes.ts` fixture are all unchanged.

### Diff

```
 src/__tests__/drift/logic-pin.test.ts |  4 ++--
 src/__tests__/drift/model-registry.ts | 12 +++++++++---
 2 files changed, 11 insertions(+), 5 deletions(-)
```

### Known side effect

Gemini's truncated-listing trust floor is
`includeFamilies[provider].size`, so it
drops from 11 to 9. Reversible: move the families back and re-pin.

---

## Local red-green proof

### 1. Baseline — before any edit, drift suites all green

```
$ pnpm exec vitest run src/__tests__/drift/
 ✓ src/__tests__/drift/model-registry.ts ... (all 9 files)
 Test Files  9 passed (9)
      Tests  87 passed (87)
```

### 2. RED — registry membership moved, pins NOT yet updated

`logic-pin.test.ts` fails loudly on the stale membership pins, exactly
as its
module doc promises:

```
$ pnpm exec vitest run src/__tests__/drift/logic-pin.test.ts

 × classification-data membership freeze > freezes includeFamilies.gemini membership
   → Frozen data set "includeFamilies.gemini" membership changed (now:
     ["gemini-2.0-flash","gemini-2.0-flash-lite","gemini-2.5-flash",
      "gemini-2.5-flash-lite","gemini-2.5-pro","gemini-3.1-flash-lite",
      "gemini-3.5-flash","gemini-3.5-flash-lite","gemini-3.6-flash"]).
     Expected: "4a9428b64ffcff0fbb79878d88ed993ffac53e43d64e26bcf5d86509626f593d"
     Received: "c2e2c56b8f8d5fc56152b4633e7d3782e95b7eeb9bc123da71f00e884a54a743"

 × classification-data membership freeze > freezes excludeFamilies.gemini membership
   → Frozen data set "excludeFamilies.gemini" membership changed (now:
     ["aqa","gemini-1.5-flash","gemini-1.5-pro","gemini-2.0-flash-exp", ...]).
     Expected: "e3545138234ad782937f66760bef942fca7d4bd0934a87da30bf6e5816ba69b1"
     Received: "c95dedab7588212bbba0bf9ab6434bfd43a449adbe7b35f81f48271ee849a9c2"

 Test Files  1 failed (1)
      Tests  2 failed | 18 passed (20)
```

The two new pin values were derived independently —
`sha256(JSON.stringify([...set].sort()))`
recomputed from `model-registry.ts` in a standalone script, matching the
values the
test itself reports:

```
includeFamilies.gemini c2e2c56b8f8d5fc56152b4633e7d3782e95b7eeb9bc123da71f00e884a54a743
excludeFamilies.gemini c95dedab7588212bbba0bf9ab6434bfd43a449adbe7b35f81f48271ee849a9c2
```

### 3. GREEN — pins re-pinned, same command

```
$ pnpm exec vitest run src/__tests__/drift/logic-pin.test.ts
 ✓ src/__tests__/drift/logic-pin.test.ts (20 tests) 3ms
 Test Files  1 passed (1)
      Tests  20 passed (20)
```

### 4. Full suite — zero regressions

```
$ pnpm test
 Test Files  165 passed | 1 skipped (166)
      Tests  4724 passed | 47 skipped (4771)
```

And the suites that specifically pin `gemini-1.5-*` behavior:

```
$ pnpm exec vitest run src/__tests__/model-utils.test.ts \
    src/__tests__/gemini-reasoning-capability.test.ts \
    src/__tests__/gemini-audio-reasoning-capability.test.ts \
    src/__tests__/gemini-toolonly-reasoning-capability.test.ts \
    src/__tests__/gemini.test.ts src/__tests__/vertex-ai.test.ts \
    src/__tests__/drift/ src/__tests__/drift-sync-core.test.ts
 Test Files  16 passed (16)
      Tests  236 passed (236)
```

### 5. Drift resolution proved offline

`scripts/drift-sync.ts` has no `--dry-run` / `--check` flag, so the
canary was
exercised directly instead: `detectDeprecatedFamiliesForSync` is
exported and
pure, so it was called against a simulated live `/models` listing
containing every
gemini family aimock classifies **except** `gemini-1.5-pro` /
`gemini-1.5-flash`
(31 ids — comfortably over the trust floor under both the old size 11
and the new
size 9), once on the pre-fix tree and once on the fixed tree.

Pre-fix (this branch at `85fc683`) — reproduces the needs-human routing:

```
includeFamilies.gemini.size (trust floor) = 11
simulated live listing length            = 31
status = checked
candidates = [{"provider":"gemini","family":"gemini-1.5-flash","stillReferenced":true},
              {"provider":"gemini","family":"gemini-1.5-pro","stillReferenced":true}]
gemini-1.5 candidates    = 2
needs-human candidates   = 2
>>> STILL FLAGGED (red)
```

Post-fix (`a3dc250`) — resolved:

```
includeFamilies.gemini.size (trust floor) = 9
simulated live listing length            = 31
status = checked
candidates = []
gemini-1.5 candidates    = 0
needs-human candidates   = 0
>>> RESOLVED: gemini-1.5 NOT flagged
```

`detectDeprecatedFamiliesForSync` is the exact function `drift-sync.ts`
uses to build
the deprecation candidate set, so this is the real routing surface, not
a stand-in.

**Plus a genuine LIVE confirmation from this PR's own CI.** The
`drift-live-pr` check
ran the drift suite against the real provider `/models` listings on this
head commit
(`preflight: Gemini key OK`) and the uploaded head drift report contains
exactly one
entry — a pre-existing *anthropic* `claude-opus-5` unclassified-family
finding, also
present in the base report — with **zero** `gemini-1.5` mentions and no
gemini entry
at all:

```
Drift gate PASSED: 1 advisory (pre-existing) finding(s) surfaced, none new-in-head.
```

That confirms against the live listing that moving both families to
`excludeFamilies` introduced no new drift and no unclassified-family
regression
(`isClassifiedFamily` is satisfied by include **or** exclude). To be
precise about
what it does *not* prove: this leg does not surface the
still-referenced-deprecation
class as a report entry — the base report shows zero `gemini-1.5`
mentions too — so
the live job is regression evidence, and the needs-human routing fix
itself is
proved by the `detectDeprecatedFamiliesForSync` red-green above.

### 6. Pre-push gates

```
$ pnpm format:check   → All matched files use Prettier code style!
$ pnpm lint           → clean (no output)
$ pnpm exec tsc --noEmit → clean (no output)
$ pnpm build          → ✔ Build complete (273 files)
$ pnpm test:exports   → No problems found 🌟
```

### 7. Zero behavior change → no version bump, no changeset

The test that matters is not "which directory did the files sit in" but
**does this
change aimock's behavior for a consumer?** It does not, and that is
verified three
independent ways.

**(a) The registry is not reachable from shipped code.** Every importer
of
`src/__tests__/drift/model-registry.ts` is a test or the maintenance
script:

```
scripts/drift-sync.ts                              (repo script, not published)
src/__tests__/drift-sync-core.test.ts
src/__tests__/drift-sync-mirror-equivalence.test.ts
src/__tests__/drift/logic-pin.test.ts
src/__tests__/drift/model-registry.test.ts
src/__tests__/drift/models.drift.ts
```

No shipped-source importer exists — `src/model-utils.ts`,
`src/server.ts`, the
handlers and every published entrypoint in `tsdown.config.ts` are all
absent from
that list. `includeFamilies` / `excludeFamilies` are drift-detection
*metadata*, not
a runtime allowlist. Confirmed at the artifact level too: `dist/`
contains zero
occurrences of `includeFamilies`, `excludeFamilies` or
`isClassifiedFamily`, so the
data isn't inlined either, and `npm pack --dry-run` ships 576 files with
**none**
matching `__tests__` / `/drift/` / `model-registry` / `logic-pin` /
`src/`
(`files` allowlist is
`["dist","fixtures",".claude-plugin","skills","CHANGELOG.md"]`;
`scripts/` ships 0 entries; no `.npmignore`).

**(b) The published artifact is byte-identical.** Built `dist/` from the
pre-fix and
post-fix trees and compared all 285 shipped code/declaration files. Note
the build
is **not** reproducible — two builds of *identical* source differ in
declaration
sourcemaps and in tsdown's non-deterministic import-alias numbering
(`node_http0` ↔ `node_http1`), so a naive whole-dist hash gives a false
positive.
Normalizing that noise, with two identical-source builds as the control
noise floor:

```
control: two IDENTICAL-source builds   → files=285 differing=0
subject: PRE-fix vs POST-fix build     → files=285 differing=0
VERDICT: PUBLISHED ARTIFACT SEMANTICALLY IDENTICAL
```

**(c) Consumer-visible behavior is bit-identical.** Probed the *built*
`dist/` before
and after with real requests for both ids — `generateContent`, SSE
`streamGenerateContent`, strict-on and strict-off reasoning, the
OpenAI-compatible
`/v1/chat/completions` surface, and `/v1/models` — plus the
`isReasoningModel()`
verdicts and the captured capability warnings. The two JSON captures
compare **equal**:

```
PRE  isReasoningModel: {'gemini-1.5-pro': False, 'gemini-1.5-flash': False}
POST isReasoningModel: {'gemini-1.5-pro': False, 'gemini-1.5-flash': False}
controls (unchanged): gemini-2.5-pro=True, gemini-pro=True, some-future-model=True
8/8 request probes identical (all HTTP 200, identical bodies)
capability warnings identical, still naming both gemini-1.5 ids
identical overall: True
```

So aimock still mocks both ids identically, `isReasoningModel()` still
returns
`false` for both, and the advertised model list is unchanged.

**Release mechanism.** `publish-release.yml` triggers on push to `main`,
compares
`package.json`'s version against npm, and publishes + tags only when
that version is
not yet published — i.e. releases come from a **manual `package.json`
version bump**.
There is no Changesets setup (no `.changeset/`, no `@changesets/cli`),
no
release-please and no semantic-release, and no workflow derives a
version from commit
messages, so **the conventional-commit type does not trigger a
release**; the
`fix(drift):` prefix here matches the repo's own prior art for registry
classification changes (#315, #322) and is inert with respect to
versioning.
`publish-commit.yml` → `pkg-pr-new` is just the per-PR preview publish
behind the
"Continuous Releases" / "preview" checks and gates nothing on
versioning. **No
changeset file and no version bump are required.**

Minor pre-existing caveat: `unreleased-check.yml` counts commits
touching `src/`
since the last tag without excluding `src/__tests__/`, so on merge it
will emit a
`::warning::` that the version wasn't bumped. It is a warning only — no
`exit 1` —
so it does not fail CI. #336 had the same shape.

---

## Separate latent defect, filed for follow-up (NOT introduced by this
PR)

The *mechanical* registry-edit path documented for new-family notes is
currently
dead-locked by its own gate, so `ok-applied` can never happen: gate-2 of
`drift-sync-check.ts` re-runs `logic-pin.test.ts` as a subprocess, that
file's
`DATA_FROZEN` block freezes set **membership**, and a mechanical
add/remove *is* a
membership change — so it trips the pin, gets reverted by `revertFiles`,
and
reports `gate-failed`. #335 shipped the mechanical path; #336 added
`DATA_FROZEN`
40 minutes later. It has never been observed because no `ok-applied` run
has ever
occurred. The `ok-applied` test injects a fake gate (`makeFakeDeps`), so
it asserts
`runSyncCheck` was *called*, never that the real pin would pass. Worth
fixing
separately — either exempt the `DATA_FROZEN` membership hashes from
gate-2, or have
drift-sync re-pin the hash as part of the allowlisted data edit and lean
on human
PR review as the guard.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant